-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle issues introduced by OTLP gRPC protocol #170
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In this commit, we are handling issues that arise from gRPC. Essentially, if we build gRPC artifacts into our Docker image, it causes the Docker image to only be compatible with applications built using the same Python version. To solve this, we are doing two things: 1) we are removing gRPC artifacts from the docker image and 2) we are changing the default OTLP protocol to be HTTP.
thpierce
commented
Apr 22, 2024
srprash
reviewed
Apr 22, 2024
aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_configurator.py
Outdated
Show resolved
Hide resolved
aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_configurator.py
Show resolved
Hide resolved
aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_distro.py
Show resolved
Hide resolved
mxiamxia
reviewed
Apr 22, 2024
aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_distro.py
Show resolved
Hide resolved
mxiamxia
approved these changes
Apr 22, 2024
srprash
approved these changes
Apr 22, 2024
mxiamxia
approved these changes
Apr 22, 2024
thpierce
added a commit
that referenced
this pull request
Apr 22, 2024
In this commit, we are handling issues that arise from gRPC. Essentially, if we build gRPC artifacts into our Docker image, it causes the Docker image to only be compatible with applications built using the same Python version. To solve this, we are doing two things: 1) we are removing gRPC artifacts from the docker image and 2) we are changing the default OTLP protocol to be HTTP. If customers attempt to set the protocol as gRPC for ApplicationSignals, we will set the default endpoint correctly. Also we are changing Docker image to build with Python 3.11, which is what we were originally doing when we encountered this issue (reference: 5b3ed74). This is what the upstream does (see [autoinstrumentation/python/Dockerfile](https://github.com/open-telemetry/opentelemetry-operator/blob/b5bb0ae34720d4be2d229dafecb87b61b37699b0/autoinstrumentation/python/Dockerfile)), and having parity here is beneficial to us. Testing: * Create `app.py`: ``` from time import sleep import boto3 try: boto3.client('s3').list_buckets() except Exception: sleep(100) ``` * Run `./scripts/build_and_install_distro.sh` * Run: ``` export OTEL_PYTHON_DISTRO="aws_distro" export OTEL_PYTHON_CONFIGURATOR="aws_configurator" export OTEL_METRICS_EXPORTER="none" unset OTEL_EXPORTER_OTLP_PROTOCOL unset OTEL_AWS_APPLICATION_SIGNALS_ENABLED ``` * Run `opentelemetry-instrument python ./app.py` ``` urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc647cd5e50>: Failed to establish a new connection: [Errno 111] Connection refused')) ``` * Run `export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; opentelemetry-instrument python ./app.py` ``` requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f20ac96e490>: Failed to establish a new connection: [Errno 111] Connection refused')) ``` * Run `export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument python ./app.py` ``` Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s. ``` * Run ``` unset OTEL_EXPORTER_OTLP_PROTOCOL export OTEL_METRIC_EXPORT_INTERVAL=1000 export OTEL_AWS_APPLICATION_SIGNALS_ENABLED=True ``` * Run `opentelemetry-instrument python ./app.py` ``` urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4316): Max retries exceeded with url: /v1/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8f6c5aa5b0>: Failed to establish a new connection: [Errno 111] Connection refused')) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8f6dd9c9d0>: Failed to establish a new connection: [Errno 111] Connection refused')) ``` * Run `export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; opentelemetry-instrument python ./app.py` ``` urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5868efdc0>: Failed to establish a new connection: [Errno 111] Connection refused')) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5868efdc0>: Failed to establish a new connection: [Errno 111] Connection refused')) ``` * Run `export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument python ./app.py` ``` Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4315, retrying in 1s. Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s. ``` By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
thpierce
added a commit
that referenced
this pull request
Apr 22, 2024
In this PR, we are bringing in patches to resolve an issue introduced by depending on OTLP gRPC protocol. Please see #170 for more details. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this commit, we are handling issues that arise from gRPC. Essentially, if we build gRPC artifacts into our Docker image, it causes the Docker image to only be compatible with applications built using the same Python version. To solve this, we are doing two things: 1) we are removing gRPC artifacts from the docker image and 2) we are changing the default OTLP protocol to be HTTP. If customers attempt to set the protocol as gRPC for ApplicationSignals, we will set the default endpoint correctly.
Also we are changing Docker image to build with Python 3.11, which is what we were originally doing when we encountered this issue (reference: 5b3ed74). This is what the upstream does (see autoinstrumentation/python/Dockerfile), and having parity here is beneficial to us.
Testing:
app.py
:./scripts/build_and_install_distro.sh
opentelemetry-instrument python ./app.py
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; opentelemetry-instrument python ./app.py
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument python ./app.py
opentelemetry-instrument python ./app.py
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; opentelemetry-instrument python ./app.py
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument python ./app.py
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.